DOCUMENT RESUME 



ED 376 902 
TITLE 

INSTITUTION 

REPORT NO 
PUB DATE 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



JC 950 009 

Evaluating Institutional Policies on Program 
Evaluation. General Guidel ines . 

Quebec Commission on the Evaluation of Collegiate 
Teaching (Quebec) . 
2410-0515; ISBN-2-550-09724-6 
Oct 94 

28p.; For related guidelines, see ED 369 434-436. 
Guides - Non-Classroom Use (055) 

MF01/PC02 Plus Postage. 

Community Colleges; ^Evaluation Criteria; *Evaluat i on 
Methods; Foreign Countries; ^Institutional 
Evaluation; *Policy Formation; -'Program Evaluation; 
-Self Evaluation (Groups); Statewide Planning; Two 
Year Colleges 

''Commission d Evaluation de 1 Enseignement Coll PQ; 
Quebec 

ABSTRACT 
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devaluation de 1 ' enseignement collegial is to assess institutional 
policies on program evaluation (IPPE's), and their implementation 
frameworks. This document examines the Commission's mandate, aims and 
approach; Commission guidelines on the formulation of institutional 
policy on program evaluation; and the evaluation conditions and 
criteria governing the policy and its implementation. Part I of the 
report covers the Commission's mandate and the renewal of college 
education in Quebec; the legal framework of college program 
evaluation; the Commission's aims for evaluation IPPE's; and a 
progressive approach targeting the development of systematic 
institutional program evaluation practices. Part II provides a 
definition of an IPPE; a discussion of the conditions (i.e., ■ 
leadership, participation, and respect for ethics) and 
characteristics (i.e., relevance, feasibility, and stringency) of a 
quality evaluation. This section also describes the essential 
elements of an IPPE: (1) general policy aims and objectives; (2) 
responsibility sharing; (3) program information system; (4) method of 
identifying the programs to be evaluated; (5) program evaluation 
process; and (6) policy review mechanism. Part II concludes by 
establishing the connection between an IPPE and the evaluations 
requested by the Commission. Part III focuses on conditions and 
criteria for policy evaluation and for the evaluation of policy 
implementation. A summary of the Commission's requirements and 
suggestions is appended. (KP) 
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Introduction 



The Commission devaluation de Tenseignement coll6gial was created under Bill 83, which 
was assented to on June 15, 1993 1 . An autonomous, independent body, the Commission 
is mandated to assess institutional policies on the evaluation of student achievement 2 
(IPESAs) and institutional policies on program evaluation (IPPEs), as well as their 
respective implementation frameworks. It is also responsible for evaluating the implemen- 
tation of the programs themselves 3 , and may in some cases rule on a given college's 
competence to grant the Diploma of College Studies (DEC) 4 . 

The Commission aims to evaluate institutional policies on program evaluation by maintai- 
ning contacts established with the colleges through the publication of this document. It is 
well aware that its basic choices in terms of policy evaluation and their consequences on 
the formulation of an IPPE and its implementation will influence not only the process of 
defining these policies, but also the evaluation practices subsequently used in the colleges. 
It thus proposes a broad, flexible framework enabling each college to develop an IPPE that 
reflects the characteristics of the environment in which it will be implemented. 

This document is divided into three sections, devoted respectively to the Commission's 
mandate, aims and approach; the formulation of the policy; and the evaluation conditions 
and criteria governing the policy and its implementation. 



1. An Act resptcting the Commission devaluation de V enseignement colUgial and amending certain 
legislative provisions. Bill 83 (1993, Chapter 26). 

2. The Commission has published two sets of general guidelines on evaluating institutional policies on the 
evaluation of student achievement: Evaluating Institutional Policies on the Evaluation of Student 
Achievement. General Guidelines, Quebec, Gouvernemcnt du Qu6bec, 1994, 20 pp., and L Evaluation 
des politiques instituiionnelles devaluation des apprentissages. Cadre de reference adapti aux 
dtablissements offrant uniquement des programmes conduisant & une AEC t Quebec, Gouvemement du 
Quebec, 1994, 22 pp. 

3. The Commission has published a set of general guidelines and a general guide on program evaluation: 
Evaluating Programs of Studies. General Guidelines, Qu6bec, Gouvemement du Quebec, 1994, 14 pp.; 
and General Guide to the Evaluation of Programs of Studies by the Commission devaluation de 
V enseignement coiUgial, Quebec, Gouvemement du Quebec, 1994, 26 pp. 

4. The Commission will be publishing general guidelines on the cc npetence of colleges to award the DEC. 



Part One 

The Commission's Mandate, Aims and Approach With Regard 
to Institutional Policies on Program Evaluation 



1. The Commission's Mandate and the Renewal of College Education 

The creation of the Commission devaluation de l'enseignement coll6gial is an important 
measure contributing to the renewal of college education. It reflects the collective will of 
the Minister of Education, the colleges, and the vast majority of organizations and 
associations in educational and socioeconomic circles to recognize the importance of 
college-level institutions in higher education and provide them with the tools they need to 
develop education at this level. As a result of the renewal, colleges will assume more 
responsibilities in this regard, and internal and external evaluation mechanisms will become 
more reliable and effective in attesting to the results of this empowerment. Policy and 
program evaluation should help improve the quality of procedures and, ultimately, owing 
to the critical reflection fostered by the assessment process, raise the calibre of college 
education and make it more relevant. As the Commission fulfils the various components 
of its mandate, the value of a college education and the diplomas colleges offer should 
receive greater recognition and credibility. 

The Commission will thus contribute to the renewal of college education by conducting 
evaluations aimed at guaranteeing and attesting to the quality of student achievement and 
study programs at this level. 

2. The Legal Framework of College Program Evaluation 

The IPPE evaluation mandate is based on the legislation establishing the Commission. This 
legislation is part of the broader framework of statutes, regulations and directives which 
specify the obligations and responsibilities of the colleges and the Commission, including 
the General and Vocational Colleges Act, the Act respecting private education, the College 
Education Regulations (RREC), and the Decision of the Minister of Education and Science 
dated 6 January 1994 respecting progressive application measures of college education. 



8 



2.1 The colleges' obligations and responsibilities 



Section 24 of the RREC 5 , scheduled to take effect in fall 1995 6 , stipulates that, after 
consulting the Commission des 6tudes, colleges shall adopt an institutional policy on 
program evaluation and ensure that it is implemented. The General and Vocational Colleges 
Act stipulates that each college's Commission des 6tudes be mandated to advise the board 
of directors on all matters related to programs of studies, including "proposals for 
institutional policy on the evaluation of programs of studies," "proposals for programs of 
studies," and "the selection of learning activities that are within the jurisdiction of the 
college" 7 . 

2.2 The Commission's obligations and responsibilities 

Under its incorporating legislation, the Commission is mandated to assess each college's 
institutional policies on the evaluation of programs of studies, and how these policies are 
implemented 8 . 

The Commission may conduct an evaluation when and in the manner it deems expedient. 
It disseminates its evaluation reports as it deems appropriate, and may make recommenda- 
tions to the college concerned and to the Minister. 



3. The Commission's Aims for Evaluating Institutional Policies on 
Program Evaluation 

The following are certain general considerations governing IPPE objectives, the framework 
within which IPPEs are formulated, and the approach the Commission uses to evaluate 
them. 



5. College Education Regulations (R.S.Q., c. C-29, s. 18; 1993, c. 25, s. 11), s. 24. 

6. Decision of the Minister of Education and Science dated 6 January 1994 respecting the progressive 
application measures of college education. 

7. General and Vocational Colleges Act, sections 17.01 and 17.02. 

8. An Act respecting the Commission devaluation de I' enseignement colUgial and amending certain 
legislative provisions, s. 13, 2nd par., subpar. 2. 

4 
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3.1 IPPE objectives : providing a framework for institutional program evaluation 
practices 

Ensuring a consistently high level of program quality requires the implementation of a 
policy designed to support evaluation practices. An institutional policy on program 
evaluation constitutes a tool making it possible to adjust program implementation as 
needed, propose solutions to potential problems, and conduct in-depth program evaluation. 
In general, this policy should give colleges more control over the quality of their programs 
and show young people and all Quebeckers that they provide a "high-calibre, top-quality 
education that enables them to attain the highest possible skill standards' 9 . 

3.2 A flexible framework for formulating IPPEs 

The Commission realizes that for most colleges, formulating an IPPE is a new experience. 
Consequently, it proposes a flexible framework enabling each college to determine its own 
general aims and objectives, design a program information system, determine the appro- 
priate moment for conducting an evaluation, define its evaluation process, and decide how 
to incorporate this new evaluation task into its program management practices. This 
framework will also enable each college to add components or elements to the policy based 
on its own specific needs and practices. 

These general guidelines are designed to help each college formulate a comprehensive 
policy to be applied at its own pace based on its experience in the field, its options for 
action, and the status of its programs. 

3.3 The Commission's approach to evaluating IPPEs and their implementation 

When evaluating the text of an IPPE, the Commission will essentially try to determine 
whether the policy constitutes a valuable program evaluation tool. To do so, it will 
ascertain whether the policy includes the elements and components it considers essential 
in supporting the evaluations 10 , and it will also assess its potential effectiveness 11 . The 
Commission's recommendations and suggestions will mainly target the improvement of the 
policy in order to ensure program evaluation quality. 



9. MESS, Colleges For The 21st Century, Quebec, April 1993, p. 19. 

10. This is the scope criterion defined in Part Three of this document. 

11. This criterion is defined in Part Three of this document. 



When evaluating the implementation of an IPPE, the Commission will take into consider- 
ation the college's progressive implementation of that policy. Priority will be given to the 
creation of a program information system and its use by the college. The Commission will 
then evaluate the overall results of policy implementation. This final evaluation will 
consist essentially in ensuring that the policy is implemented effectively, that it results in 
quality evaluations, and that the college takes the necessary steps to improve program 
quality 12 . 

The Commission feels that this progressive, practical approach will result in the harmo- 
nious, efficient implementation of the new institutional policy. 

4. A Progressive Approach Targeting the Development of Systematic 
Institutional Program Evaluation Practices 

The Commission encourages the colleges to begin by formulating their policy and then 
implement it on a step-by-step basis. Colleges that are more experienced in program 
evaluation will proceed more quickly through the various phases. 

4.1 Policy formulation 

The Commission recommends that the formulation of the IPPE comprise elements inherent 
to all such policies, i.e., general aims, objectives, responsibility-sharing, and review 
mechanisms. To be functional, the policy must include the following components, essential 
to any evaluation tool : first, the implementation of a program information system based 
on a group of key indicators; second, a method of identifying the programs to be evaluated, 
enabling evaluations to be conducted as needed and including general rules governing the 
frequency of evaluations; third, the development of a program evaluation process which 
includes specific guidelines for the preparation and content of an evaluation outline, the 
evaluation per se 9 and evaluation follow-up. The IPPE must thus provide a suitable 
framework for the college's program evaluation practices. The results of its progressive 
implementation should render the policy increasingly relevant and operational as the 
necessary changes are made to its essential components and elements. 



12. These are the compliance and effectiveness criteria defined in Part Three of this document 



4.2 Progressive policy implementation 
Step J : A program information system 

In the short term, the Commission intends to focus on the creation of an information 
system enabling program implementation to be monitored and analyzed. Such a system 
comprises indicators, i.e., quantitative and qualitative data representing the "health status" 
of the programs offered by the college. Applied to a specific program, the monitoring of 
these indicators allows for immediate action, either by adapting program implementation 
or by taking steps to conduct a more comprehensive evaluation. 

During this first phase, the colleges will also participate in the Commission's program 
evaluations based on its specific guides. Colleges with operational information system." 
will find them useful in the evaluation process. In return, this exercise will enhance 
program evaluation skills. 

Step 2 : Partial policy implementation 

In the medium term, once the colleges have acquired the necessary experience, they will 
be capable of implementing the key elements of their policy. They will thus be able to use 
an operational information system and make the appropriate decisions, i.e., to adapt a 
program's implementation where the available information so indicates or, where this is not 
the case, to conduct a program evaluation based on various evaluation questions. 

The colleges will continue to participate in program evaluations conducted by the Commis- 
sion, which may also request that they use their policy to carry out certain program 
evaluations under its supervision, e.g. the evaluation of a program offered at only one 
college. 

Step 3 : Full policy implementation 

In the longer term, the Commission feels that the colleges could systematically evaluate 
their programs themselves. The choice of programs to be evaluated would then be based 
on the status of each program, the maximum interval between two in-depth evaluations, and 
various external requests made, for instance, by the Commission or an accreditation agency. 

* * * 
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By following this procedure, the Commission believes that the colleges will gradually be 
able to develop systematic program evaluation practices based on a methodic, credible 
evaluation process. Incorporated into current program management practices, this 
process will attest to their ability to ensure adequate follow-up of their programs* 
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Part Two 

Formulation of the Institutional Policy on Program Evaluation 



1. Definition of an Institutional Policy on Program Evaluation 

The Commission views the IPPE as a management tool designed to help colleges conduct 
program evaluation. 

It defines the IPPE as follows : 

An official document in which the college outlines how it assumes its 
responsibility for evaluating its programs of studies 13 and attesting to 
the results. 

This document is approved by the college's board of directors following consultation with 
its Commission des 6tudes. 



2. Conditions and Characteristics of a Quality Evaluation 

Experience acquired to date in the field of program evaluation 14 shows that leadership, 
participation, and respect for ethics are three conditions essential to a quality evaluation. 
It also shows that quality evaluations are characterized in particular by their relevance, 
feasibility, and stringency. The Commission will consider these conditions and characteris- 
tics when evaluating policies and their implementation. 



13. The RREC defines a program of studies as follows: "an integrated set of learning activities leading to 
the achievement of educational objectives based on set standards." College Education Regulations, 
Qu6bec t 1993, s. 1. 

14. Based on the Joint Committee on Standards for Educational Evaluation, an account of whose work is 
published in The Program Evaluation Standards. How to Assess Evaluations of Educational Programs, 
2nd edition, Thousand Oaks, California, Sage Publications, 1994, 222 pp. 

"1.4 



2.1 Conditions 



Leadership 

Appropriate leadership is apparent when various phases of the evaluation process are 
carried out by persons or authorities who believe in its importance and are capable of 
ensuring its success. Consequently, it is essential that the highest authorities of the college 
make a firm commitment to the principle of evaluation, incorporate it into their manage- 
ment practices, help carry it out, and ensure the necessary follow-up. It is also important 
that a program be evaluated by individuals who are directly involved in its implementation 
and who have the authority and commitment to see it through under optimum conditions. 

Participation 

A commitment to participation is apparent when the persons and authorities involved in 
the evaluation of a program participate actively in the various phases of the evaluation 
process. This participation is necessary to ensure not only that the evaluation produces the 
desired results, but also that any recommendations which may subsequently be formulated 
are clearly understood and followed. 

Among the persons and authorities involved, the participation of teachers in identifying 
problems and seeking solutions appears essential to the success of the evaluation. Other 
parties should also participate, especially students. Depending on the type of evaluation, 
the opinion of the professional and technical staff, former students, and representatives of 
the labour market and universities could be very important. 

The Commission encourages widespread participation with a view to developing a 
corporate culture capable of producing quality program evaluations and incorporating this 
evaluation task into regular program management practices. This calls for a firm commit- 
ment on the part of all those involved to seeing the evaluation process through. 

Respect for ethics 

To ensure the success of its evaluations, it is important that the college establish a code of 
ethics to guide those responsible for evaluation and those in charge of processing and 
disseminating the information obtained. Such a code clearly defines the boundaries of 
information dissemination based on both the confidential nature of personal information and 
the public's right to information. It can also prevent eventual conflicts of interest 

10 
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2.2 Characteristics 
Relevance 

Designed as a management tool for colleges, the policy must contribute to a relevant 
evaluation, i.e., one which is capable of clearly establishing program status, finding 
concrete solutions to the problems observed, and helping improve education quality. Only 
after the evaluation is completed will it be possible to determine its relevance by assessing 
its impact on the implementation of the program evaluated, on those involved in the 
implementation, and on the college itself. It is nonetheless important to ensure before the 
process begins that all conditions favour a relevant evaluation. 

Feasibility 

Evaluation feasibility is based on a realistic evaluation, a simple, appropriate methodology, 
a climate of trust, and consideration of the various interests of the persons and authorities 
involved. To be feasible, the evaluation process must be easily applicable and efficient, 
and take into account the input of those involved. The choice of evaluation questions must 
be based on the resources available and subsequent action options. In short, feasibility 
implies that the college's policy will generate an efficient evaluation process based on the 
time and resources available, the interests of those involved, and options for action. 

Stringency 

The stringency of the evaluation process concerns the quality of the information and the 
approach used to obtain it. This information must be technically accurate and contribute 
to subsequent decision-making. If decisions related to evaluation results are to be credible, 
they must be based on a logical sequence beginning with the evaluation questions, 
proceeding to the analysis and interpretation of the data gathered, and ending with 
conclusions and recommendations. Evaluation questions must be clear and specific, the 
data accurate and valid, the analyses explicit, and the recommendations relevant. This 
stringency of the evaluation process and methods must contribute to producing valid, 
reliable, and credible evaluations. 



11 



3. Essential Components of an IPPE 



For program evaluation to be efficient, the Commission believes that any policy should 
include the following essential components or their equivalent : 

- General policy aims and objectives 

- Responsibility-sharing 

- Program information system 

- Method of identifying the programs to be evaluated 

- Program evaluation process 

• Evaluation outline 

• Evaluation 

• Evaluation follow-up 

- Policy review mechanism 

The Commission indicates the elements defining or characterizing each of the components. 
It also suggests other elements likely to contribute to formulating the policy 15 , 

3.1 General policy aims and objectives 

The general aims express the principles, values and orientations which determine the 
fundamental priorities of the policy as defined in the objectives. They guide the college 
in conducting the evaluation and implementing its results. 

The Commission feels that the principal aim of any program evaluation policy should be 
to ensure ongoing improvement of the quality of education offered to the students. 

The objectives express the goals of, and the results expected from, the implementation of 
the policy and the resulting program evaluations 16 . Formulated clearly, precisely, and 
concretely, they are measurable and focus on achieving the other policy components, 
usually within a predetermined time frame. 



15. To facilitate formulation of the IPPE, the Commission has summarized its requirements and suggestions 
in an appendix. Some examples are also provided to clarify its position. 

16. In general, the policy makes a distinction between the objectives pertaining to its implementation and 
the general objectives targeted by the program evaluations. Of course, each program evaluation targets 
specific objectives that fall within the general objectives defined in the policy. 
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3.2 Responsibility-sharing 



The policy provides for a clearly defined formula of responsibility-sharing with regard to 
the program information system, the method of identifying the programs to be evaluated, 
and the program evaluation process. Responsibilities are normally assumed by a number 
of individuals, committees, or administrative and academic bodies, i.e., the board of 
directors, the Commission des etudes, program administrators, program committees, 
departments, teachers, technical and professional staff, students, etc. 

The policy identifies an authority (individual or committee) in charge of implementing the 
evaluation policy. A second authority" is appointed to conduct program evaluation; if 
this authority is a committee, the policy may include general rules governing its makeup. 

3.3 Program information system 

Monitoring is an essential element of program management. So that this element may be 
carried out, the policy provides for the creation of a program information system and sets 
out its components and operating conditions 18 . This system includes indicators and 
data 19 which pertain to enrolment, progression through school, the success rate, teachers' 
and students' perception of the program, job placement, employers' assessment, university 
admission, and the success rate for university studies, and which contribute to monitonng 
program development. 

In addition to enabling immediate corrections to be made to any flaws detected, this 
information system, which could be described as a "program control panel," also constitutes 
a decisive element in a systematic program evaluation practice. Along with other 
organizational or cyclical factors, a periodic review of information system data and 



17. In general, two separate authorities are appointed for program evaluation and policy implementation. 
However, the same authority may assume both roles. 

18. The Commission will pay special attention to the description and operating conditions of this component 
when evaluating the text of the policy. 

19 Most colleges already use a more or less developed information system in their program management. 
btfdSf to toSbLtton it gathers itself, the college may use data from sources such as the 
MinSde TEducauon, including documents produced using the Systeme d'information et de gestion 
<£to£Z sufl'effecttf coll6giM (SIGDEQ and the Systeme d'information sur tes chemmerrients 
SlaS^SgS^ESCO? or from the Service regional d'admission du ^^^"P*™ 
(SRAM), including publications based on the Profil scolaire des etudiants par programme (PSEP) The 
coUeges will be able to follow the progression of their former students with the university student 
inventory (RECU) system, thanks to the harmonization of the student s permanent coae. 
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indicators will help determine the appropriate moment for conducting a program evaluation. 
It will also help determine the questions to be included in the evaluation oudine. 



3.4 Method of identifying the programs to be evaluated 

The policy sets out the rules for using the "control panel" to identify which programs 
should be evaluated. It also determines when evaluations should take place, for example, 
a few years after the implementation of a new program or of major changes in an existing 
program; or when a program has not been evaluated for the maximum interval established 
by the college. 

3.5 Program evaluation process 

The policy outlines how a program is evaluated, from the preparation and approval of an 
evaluation outiine to the production of an evaluation, report and the assessment of its 
results. 

To facilitate the task of the evaluation committees and ensure a certain uniformity in the 
evaluation process, the policy defines the criteria to be used to assess the main characteris- 
tics of a program 20 . 

Evaluation outline 

The policy describes the method for preparing an evaluation outline and the administrative 
phases it must go through before it is approved. In general, preparation of the outline is 
based on an analysis done to specify the nature and scope of the evaluation to be carried 
out. Thus, the data and indicators in the program information system can contribute to 
establishing the program status and defining a number of priority factors which should 
undergo an in-depth review during program evaluation. Approval of the oudine by college 
authorities allows them to exercise the necessary leadership to conduct the evaluation and 
incorporate the results into program management and the college's strategic plans. 

The policy determines what should be included in a typical evaluation outline. It includes 
a description of the program status and the list of topics to be studied. 
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20. Colleges may use the criteria described by the Commission in its General Guide to (he Evaluation of 
£SXo2S5^SSsft! %g mUSSi0n Muation * renseignement coMgial, Quebec, Gouverne- 
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Evaluation 



The policy defines the conditions for conducting program evaluation and completing an 
evaluation outline. These conditions may pertain to the preparation and validation of data 
collection tools, the participation of individuals and college authorities and, where 
necessary, referral to external experts, as well as the progress of the report from its 
preparation and completion to its approval and dissemination. These conditions comprise 
the typical content of an evaluation report. 

Evaluation follow-up 

The policy establishes links between program evaluation and decisions made by the college 
in this regard by explaining the conditions governing the preparation and adoption of an 
action plan by college authorities. This plan translates the recommendations made pursuant 
to evaluation into action by the administrative units in charge of the program. 

The policy also sets out rules governing the dissemination of results, which must respect 
the confidential nature of the personal information contained in the evaluation reports, 

3,6 Policy review mechanism 

The importance of program evaluation as well as the perspective of progressive policy 
implementation call for a review mechanism whereby changes may be made as needed. 
These changes are approved by the college's board of directors after consultation with the 
Commission des dtudes. 



4, Connection With the Evaluations Requested by the Commission 

The essential components of the policy can also serve as a framework for program 
evaluations requested by the Commission, especially for the preparation of self-evaluation 
reports. If the college deems it useful, the policy may provide for specific conditions or 
the adaptation of existing conditions to the context of the evaluation of a program by the 
Commission. These conditions may concern, for example, the "responsibility-sharing" and 
"evaluation process" components. 
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Part Three 

Evaluation Conditions and Criteria Governing an IPPE and Its 
Implementation 



This third section sets out the conditions and criteria set by the Commission for each 
evaluation pha°e, i.e., evaluation of the policy and evaluation of its implementation. 

1. Evaluating Institutional Policies on Program Evaluation 

1.1 Policy evaluation conditions 

The Commission will evaluate the text for the components and elements it considers 
essential and assess the policy's potential to serve as an effective framework for its users, 
especially in helping them create a useful program information system, conduct quality 
program evaluations, and incorporate the evaluation task into the college's program 
management practices. 

1.2 Policy evaluation criteria 

The Commission has established two criteria for evaluating the policy : the IPPE's scope 
and its potential effectiveness. 

• Scope 

This criterion enables the Commission to ascertain whether the IPPE contains all the 
components and elements it has deemed essential. 

• Potential effectiveness 

This criterion enables the Commission to assess the policy's potential to serve as an 
effective framework for its users, allowing them to benefit from an efficient program 
information system and conduct quality evaluations, i.e., evaluations characterized by 
their relevance, feasibility, and stringency. It also enables the Commission to assess the 
validity of the mechanisms created to ensure that evaluation results are incorporated into 
program management practices. 
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1.3 The Commission's decision on the policy 

Once it has completed its analysis of the policy, the Commission makes one of the 
following decisions : 

1) The policy is satisfactory. All the critena have been met and the Commission feels that 
the description of the components and elements contained in the policy should contri- 
bute to guaranteeing the quality of program evaluations and the incorporation of the 
resrlts into program management practices. The Commission may suggest improve- 
ments to the college where it deems such suggestions relevant. The college is not 
obliged to follow up on these suggestions, but the Commission would like to know 
whether it plans to do so. 

2) The policy is partially satisfactory. Essential elements or components are absent or 
incomplete and consequently, the policy has shortcomings which may significantly 
affect the quality of program evaluations and their follow-up. Changes are thus 
necessary, and the Commission includes with its decision recommendations on which 
the college must follow up. 

3) The policy is unsatisfactory. Numerous essential components or elements are absent or 
incomplete. In such a case, the quality of the program evaluations cannot be guaran- 
teed. The policy must be reviewed and submitted to the Commission again. 

1.4 The policy evaluation report 

The Commission approves the policy evaluation report and then forwards it to the college 
and to the Minister. It makes the results of this report public as it deems appropriate. 

2, Evaluating the Implementation of the Institutional Policies on 
Program Evaluation 

2.1 Conditions for evaluating policy implementation 

To conduct a comprehensive evaluation of a college's implementation of an IPPE, the 
Commission will use the following elements : a self-evaluation report written by the 
college at its request, the reports on program evaluation conducted as part of the policy and 



18 



22 



their follow-up, and the evaluation of one or more components of the IPPE that the 
Commission may eventually conduct as part of its program evaluation in each college. 

When appropriate, the Commission will specify the content of the self-evaluation report, 
which may include evaluation by the college of the creation of the program information 
system, the method of identifying the programs to be evaluated, the evaluation process per 
se, and implementation of the resulting action plans. 

By studying the program evaluation reports, the Commission intends to determine whether 
the college is implementing its policy efficiently and whether its evaluations are likely to 
result in concrete progress. 

Analysis of these two elements, along with the results of the evaluations already conducted 
on one or more components of the IPPE, may result in a visit to the college concerned. 
Such a visit enables Commission members to meet directly with those involved in the 
evaluation process, better understand policy implementation strengths and weaknesses, and 
suggest or recommend appropriate measures to improve implementation results. 

2.2 Evaluation criteria for policy implementation 

The Commission has established two criteria for evaluating policy implementation : 
compliance and effectiveness. 

• Compliance 

This criterion enables the Commission to evaluate the implementation of the essential 
components and elements provided for in the policy. It expresses the extent to which 
the implementation of the essential components and elements corresponds to their 
description in the text. Where necessary, evaluation of the implementation shall take into 
account the changes made by the college since the policy was evaluated by the Commis- 



sion. 



. Effectiveness 

Evaluation of the effectiveness of the policy implementation enables the Commission to 
determine the extent to which the policy helps ensure the quality of program evaluation. 
More specifically, it aims to determine how the implementation of the components and 
elements described in the policy contributes to quality program evaluations (characterized 
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by their relevance, feasibility and stringency), ensures the incorporation of these results 
into program management practices, and improves the programs evaluated. 

2.3 The Commission's decision on policy implementation 

Once it has completed its analysis of policy implementation, the Commission makes one 
of the following decisions : 

1) Policy implementation has satisfactory results. All criteria have been met and the 
Commission feels that the program evaluations conducted comply with the description 
of the policy's essential elements and components and that they are high-quality, i.e., 
relevant, feasible, and stringent. In addition, the recommendations issued based on 
program evaluation results have been incorporated into program management practices 
in the form of action plans approved by the college's board of directors and implemen- 
ted by the appropriate authorities. Finally, these evaluations are effective in the sense 
that their results contribute to improving the programs. The Commission may suggest 
improvements to the college where it deems such suggestions relevant. The college is 
not obliged to follow up on these suggestions, but the Commission would like to know 
whether it plans to do so. 

2) Policy implementation has partially satisfactory results. The implementation of certain 
components and elements essential to the policy reflects shortcomings which are 
particularly obvious in the quality of the evaluations conducted or the incorporation of 
their results. The Commission thus includes with its decision recommendations which 
the college is obliged to follow up on in its subsequent program evaluations. The 
Commission may also suggest improvements to the college where it deems such 
suggestions relevant. 

3) Policy implementation has unsatisfactory results. The implementation as it is carried 
out significantly jeopardizes the quality of program evaluations or does not ensure 
incorporation of their results, and therefore does not help improve program quality. The 
Commission therefore proposes, in the form of recommendations and suggestions, 
appropriate measures to remedy the situation. 

2.4 The evaluation report as it pertains to policy implementation 

Based on its analysis of policy implementation, the Commission indicates one of the three 
above-mention-H decisions in a draft report and forwards a copy to the college. Depending 

20 



ERIC 



24 



on the decision made, the report stresses the strengths and, where applicable, the weak- 
nesses. It includes suggestions or recommendations to improve or remedy policy 
implementation. Indications on how the college should follow up on this decision may be 
provided. 

The college then has a given amount of time to submit its comments to the Commission 
and, where applicable, inform it of what it has done or plans to do to enhance the quality 
of policy implementation. 

After receiving the college's input, the Commission prepares and adopts its final evaluation 
report on policy implementation and forwards a copy to the college and to the Minister. 
The Commission makes the results of this report public as it deems appropriate. 
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Appendix 

Sum lary of the Commission's requirements and suggestions 



Components 


Required elements 


Suggested elements 


1. General aims 
and objectives 


Specify the general aims of the policy. 

Specify the objectives of the policy. 

opeciiy the general oujeviivei uugvieu uy ui^ 
program evaluations. 




2. Responsibility- 
sharing 


Appoint an authority (individual or cornmittce) 
responsible for implementing and reviewing 
the TFPE. 

Appoint an authority or person responsible 
for : 

- the program information system; 

- the method of identifying the programs to be 
evaluated. 

A*««-»s\.«nf n « a ii tVi/"*i*iti/ i"<**JT^On C1r\lf* Tor tnG 

Appoint an auuiuniy ic&^wn&iui^ *ui uiv 
program evaluation process. 


If the authority is a committee, provide 
for general rules governing its makeup 
so as to ensure the participation of those 
concerned, especially the teachers. 


3. Program 
information 
system 


Describe the components of the program 
information system by specifying the types or 
categories ot data ano lnuicaiors usui vj 
monitor the implementation of each program 
and assess the results. 

Specify the operating conditions of the 
information system. 


Provide for data and indicators pertaining 
to enrolment, progression through school, 
the success rate, teachers 1 and students' 
perception of the program, job 
placement, employers' assessment, 
university admission, and the success 
rate for university studies, 

Specify, for example, the conditions 
governing data management and the 
production and dissemination of reports 
on program status. 


4. Method of 
identifying the 
programs to be 
evaluated 


Deterrnine a method of identifying the 
programs to be evaluated including general ^ 
rules governing the use of the "control panel." 

Provide for general rules governing the 
frequency of evaluations. 




5. Program 
evaluation 
process 


Describe a program evaluation process 
including the following components : 

1) preparation and approval of an evaluation 
outline; 

2) evaluation; 

3) evaluation follow-up. 

Specify the criteria used to assess the 
program's main characteristics. 
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BEST COPY AVAILABLE 



Components 


Required elements 


Suggested elements 


5*1 Preparation 
and approval of 
the evaluation 
outline 


Describe the method for preparing an 
evaluation outline and the administrative 
phases it must go through before it is 
approved. 

Specify the content of a typical evaluation 
outline, including a description of the program 
status and the list of topics to be studied. 


Establish a link with the program 
information system. 

Typical outline content may include the 
following : the criteria on which program 
assessment is based; indications on data 
collection, analysis, and interpretation 
methods; and an estimate of the 
resources required. 


S2 Conducting the 
evaluation 


Specify the conditions for conducting an 
evaluation. 

Describe the content of a typical evaluation 
report 


Include details on data collection, 
analysis, and interpretation; preparation 
and completion of the report through to 
its adoption and dissemination; 
participation of college authorities and 
individuals involved, including teachers; 
and, where applicable, referral to external 
experts. 

A typical evaluation report may include 
the following : program description, a 
description of the evaluation process, the 
data on which the evaluation is based, 
and the resulting conclusions and 
recommendations. 


5.3 Evaluation 
follow-up 


Specify the conditions for the preparation and 
approval by college authorities of a plan of 
action translating the recommendations made 
pursuant to evaluation into action to be taken 
by the administrative units in charge of the 
program. 

Specify the rules governing dissemination of 
the results. 




6. Policy review 
mechanism 


Describe the periodic policy review 
mechanism. 

Point out that the changes are approved by the 
board of directors after consultation with the 
Commission des etudes. 




7. Connection with 
the evaluations 
conducted by the 
Commission 




Describe the specific conditions which 
serve as a framework for the evaluations 
conducted by the Commission. 
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